AsyncAPI 3.x: the fitness, and --problemType ASYNCAPI runs - #1753
LautaroPetaccio wants to merge 12 commits into
Conversation
8347c38 to
eea6cba
Compare
bf16b4d to
02bd557
Compare
0cfc38c to
a3b119e
Compare
a3b119e to
23a9db1
Compare
23a9db1 to
f87430f
Compare
f87430f to
0341ff5
Compare
| fv.updateTarget(idMapper.handleLocalTarget(fault), 1.0, index) | ||
| } | ||
|
|
||
| AsyncApiOutcome.PUBLISHED, AsyncApiOutcome.PUBLISH_FAILED -> Unit |
There was a problem hiding this comment.
what if none of the AsyncApiOutcome matches?
There was a problem hiding this comment.
They must all match, as the compiler will not compile otherwise. Did you mean that or were you talking about a reply not matching?
0341ff5 to
5051e3b
Compare
9848d77 to
a7856ff
Compare
|
I forgot to include the replies which were not matched with the schema as possible targets. Now they're included. |
72c493c to
b961897
Compare
b961897 to
aec5186
Compare
aec5186 to
315c227
Compare
| * Deliberately not drawn from [randomness]: a run repeated under the same seed would reuse | ||
| * the ids it used before, which is the one case this exists to tell apart. | ||
| */ | ||
| private val runId: String = UUID.randomUUID().toString().take(8) |
There was a problem hiding this comment.
This is problematic since it might lead to different behaviour even when the same -seed is introduced, right?
There was a problem hiding this comment.
You're right! Great catch. I've changed it to use randomness which takes into consideration the seed.
RemoteController.executeNewAsyncApiActionAndGetReply PUTs the action to the driver and reads back an AsyncApiReplyDto, as its RPC counterpart does. It has a default that throws, so the controllers that never publish need not say so one by one. EMConfig gains asyncApiReplyTimeoutMs, experimental, and a constraint: there is no test writer for AsyncAPI yet, so a run must say --createTests false rather than search for an hour and fail at the end. The two tests that already parsed --problemType ASYNCAPI say so now. Two experimental fault categories: a promised reply that never arrives, and a reply matching none of the messages the contract declares.
AsyncApiBlackBoxFitness publishes each message of a test through the driver and turns what comes back into targets: for every operation, what publishing to it was seen to do, and, when a reply came back, which of the messages the contract declares for the reply it was. A contract listing a result and an error thus gives the search two things to reach, which is the AsyncAPI analogue of REST's (status x endpoint). AsyncApiReplyClassifier is what recognises a reply: a structural match against each declared payload schema, reading the parts of JSON Schema that tell one message from another and giving the benefit of the doubt on the rest. The most specific match wins. AsyncApiModule binds it all, with the driver bound unconditionally, and Main uses it in place of the message it showed until now. One test runs a whole MIO search against a stand-in for the NCS service and sees both declared replies of an operation covered.
Review: with room for exactly two messages, a one-message test was mutated by removal, leaving a test that publishes nothing; and a test could never grow to the maximum the user allowed, only to one less. Both bounds are now stated as such, and AsyncApiStructureMutatorTest holds them. Seeding test cases is refused at start-up, like writing them, instead of failing inside the sampler. The whole-search suite is named after the class it exercises, AsyncApiModule, and the three suites that build the injector share how they do it.
Review: the ':' joining a target id and the '-' inside a correlation id were spelled out where used; they are constants now, and the two kinds of target id are built in one place each. The remote controller named its queryFromDatabase parameter four times over, once in the new call and three in the ones it mirrors; it is a constant now. The fake driver's one field comes before its companion.
…aults
Review findings, in order of what they cost a run.
The address a message is published to ignored protocol bindings. A Kafka
channel usually names no address and carries its topic in its binding --
microcks.yaml, already a test resource here, is exactly that shape -- so
messages went to the channel's name and reached nobody. The parser's
effectiveAddress() exists for this and was never called; when a channel
names no address at all, the binding's topic is now taken as the one
destination the document actually gives. An address left holding
{placeholders} is warned about rather than published to silently.
Deciding whether a number is an integer asked Jackson for a BigDecimal,
which throws for a value that overflows a double. Nothing between the
classifier and the search loop catches it, so one odd reply ended the run.
Three ways a valid reply was reported as a fault: a body that is empty or
not JSON, a const or enum written 1.0 against a reply carrying 1, and a
type or combinator this cannot read. A reply with nothing to match against
is now no finding at all, numbers compare by value, and what cannot be read
rejects nothing. A $ref that points into a schema is followed instead of
matching everything, which had quietly disabled the undeclared-reply oracle
for any operation written that way, and specificity counts through the
combinators so the more specific message wins.
Faults ignored isEnabledFaultCategory, so two experimental oracles fired on
a default run and --disabledOracleCodes did nothing. They are recorded on
the action result now, which is where the reports count faults from; before,
a run covered fault targets while every report said zero.
AsyncApiCallResult was the only action result not overriding matchedType.
An inferred problem type no longer dies on the test-generation constraint:
when the driver reports an AsyncAPI service, createTests is turned off with
a warning instead of throwing after the SUT has already been started.
The class is AsyncApiFitness: it is bound as the only fitness and reads
white-box coverage, so BlackBox in its name said something untrue.
The new remote-controller call was a copy of the RPC one; both now share the body that PUTs an action and reads the reply back. The structure mutator is a third copy of logic REST and RPC already share, marked with a TODO the way the GraphQL one marks the same debt. The reply-timeout description is published verbatim into options.md, where every neighbour is one clause; the reasoning behind the option belongs in the pull request, not in the user's terminal.
The guards that keep a valid reply from being reported as a fault had no tests, which is why they could be wrong. Now covered: a reply with no body, an operation whose reply declares no message, a number too large to be a decimal, a const compared by value, a reference into a schema, and the most specific match found through a combinator. Also covered: the topic a protocol binding names, the configured reply timeout, and that faults stay quiet until experimental oracles are asked for -- each of which would otherwise pass with the code deleted. The structure mutator's tracking is exercised with a real specification rather than null, so the add/remove bookkeeping the archive reads is checked. The module test asserts the bindings Main resolves, since this module deliberately inherits none. The sampler suite resets the static gene cache its siblings all reset.
The reply DTO no longer uses primitives, so a field the driver never set arrives as null rather than as a default. Kotlin maps those to platform types, so reading them as before would still compile and throw at run time. Each is now read deliberately. Whether the message was published has no safe default, so an absent answer is treated as a failure and said so in the error message, apart from a driver that answered false. Whether a reply was expected or arrived is read as not having happened when unset, which is what a fire-and-forget driver leaves blank. Correlation is the one that changes behaviour: a driver that does not track it at all reported false, which reads as the service having failed to echo the id back. That is a claim about the SUT the driver never made, so it is now recorded only when the driver actually looked.
Turning the experimental oracles off used to cost coverage rather than only reporting. A reply matching none of the declared messages registered nothing beyond the gated fault call, so it was indistinguishable from a recognised one and could be dropped when the solution was minimised. It now covers a target of its own, registered before the fault and independent of whether it is enabled, which is how a 500 is already handled for REST. Also in this round: - the correlation id prefix no longer comes from the seeded generator, which made a run repeated under the same seed reuse the very ids it exists to tell apart - an outcome that cannot reach target handling now says so by throwing rather than by a comment, and the branch stays exhaustive so that a newly added outcome fails to compile instead of falling through - why there is no separate black-box fitness, and why the reply classifier is written by hand rather than delegated to the validator already on the classpath, which reads draft-04 and so does not know const
The driver contract now names the two places a correlation id can travel with an enum rather than a pair of string constants, so the fitness sets the field with the enum and the tests compare against it.
315c227 to
5911a41
Compare
The run prefix was drawn outside the seeded generator, so two runs under the same seed published different correlation ids. Nothing in the search reads that value today, but the guarantee that a seed reproduces a run should not rest on nobody reading it, and where the contract puts the id in the payload the published bytes already differ. It is back on the generator, and the reason it had left is written down where it can be enforced: a reply published before the action was is not an answer to it, and only the driver can ensure that, by seeking to the end of the reply destination or by a fresh subscription. The reference Kafka driver already does, which is why the prefix never needed to. A test publishes the same two messages twice under one seed and compares the ids; it fails against the previous value.
| * before the gene builder sees it. A validator that reads a modern draft would replace most of | ||
| * this, at the cost of a new dependency. | ||
| */ | ||
| object AsyncApiReplyClassifier { |
There was a problem hiding this comment.
a utility function should not be under a service package (which is for module declarations and singletons)
There was a problem hiding this comment.
Moved it to org.evomaster.core.problem.asyncapi.classifier along with its tests.
| } else if (info.asyncApiProblem != null) { | ||
| config.problemType = EMConfig.ProblemType.ASYNCAPI | ||
| if (config.createTests) { | ||
| /* |
There was a problem hiding this comment.
leave a TODO message stating that till will need to be removed in the future
It is a stateless object nothing injects, sitting beside the sampler, fitness, module and mutator, which are all Guice-managed. It moves to a package of its own, with its test. Main also says now when the block that switches test generation off has to go, rather than only why it is there.
660ed2a to
10f8f96
Compare
Twelfth in the AsyncAPI stack, on top of #1752. With this one
--problemType ASYNCAPIruns a whole search, everything but writing the tests.What a search over an AsyncAPI service looks for
REST covers
(status × endpoint). There are no status codes here, so the targets are:(outcome × operation)— what publishing to an operation was seen to do:PUBLISHED(fire-and-forget),REPLIED, orNO_REPLY.(declared reply × operation)— when a reply came back, which of the messages the contract declares for that reply it was. A reply channel that lists a result and an error gives the search two things to reach, and telling them apart is what makes an asynchronous service observable from outside.(unrecognised reply × operation)— a reply matching none of the declared messages. It is a target of its own, registered independently of the fault below, so the search can still reach that behaviour when the oracles are off.Two things are faults, and both categories are
@Experimental, so they are reported only under--useExperimentalOracles: a promised reply that never arrives, and a reply that matches none of the declared messages. Each is recorded on the action result as well as the fitness, which is where the reports count faults from. A message the driver could not publish is neither — it is a broken setup, no target is registered for it, and the test stops there.Recognising a reply
AsyncApiReplyClassifiermatches the observed payload against each declared payload schema. It is deliberately a classifier, not a validator: it reads the parts of JSON Schema that tell one message from another —type(including type lists),required,const/enum,allOf/anyOf/oneOf,items,$refto component schemas — and gives the benefit of the doubt on anything it does not understand (formats, bounds, patterns), since a reply it fails to recognise is reported as a fault. When several declared messages fit, the most specific wins: the one pinning down the most top-level fields. No new dependency.The fitness
AsyncApiFitnessresolves everything the driver needs from the document — address, content type, correlation location and pointer, reply address, timeout — so the driver never reads the document. The address comes from the channel'seffectiveAddressfor the protocol, so a Kafka channel that names no address and carries its topic inbindings.kafka.topicis published to the topic rather than to the channel's name.The correlation id is
runId-counter.runIdis a random prefix rather than one drawn fromRandomness: a run repeated under the same seed would otherwise reuse the very ids the prefix exists to tell apart, which is the case it is there for. Headers travel as a map by way of the gene's own JSON printing, which is what knows which optional headers are on.Not supported yet, and said so with a one-time warning: a reply address announced inside the request (
reply.address.location) rather than fixed by the contract. Such an operation is published without waiting.Plumbing
RemoteController.executeNewAsyncApiActionAndGetReply, mirroring the RPC call. It has a default that throws, so the four other controllers (three of them test fakes) need not each say they cannot publish.EMConfig.asyncApiReplyTimeoutMs,@Experimental, default 5 s.--createTests falseis required for AsyncAPI until the test writer exists. Failing at start-up beats searching for an hour and failing at the end.TestCaseWriteris bound to the existingNoTestCaseWriter. Seeding test cases is refused at start-up too, for the same reason. When the problem type is inferred from the driver rather than asked for, test generation is switched off with a warning instead of failing after the SUT has started.AsyncApiModule, one for both modes, binding the driver unconditionally. It does not callsuper.configure(), like the RPC, GraphQL and Web modules: whatEnterpriseModulebinds there needsFitnessFunction<RestIndividual>.AsyncApiStructureMutator: one message more or one fewer. Unlike RPC's, it states its two bounds outright — a test keeps at least one message, and can grow to exactlymaxTestSize— because mirroring RPC's arithmetic emptied a one-message test whenmaxTestSizewas 2, and never let a test reach the maximum.Main'sASYNCAPIbranch binds the module instead of throwing.Testing
Four suites build the injector
Mainwould, with the driver replaced by a fake, through one sharedAsyncApiTestInjector.AsyncApiReplyClassifierTest(20): result vs error behind$refs, undeclared payloads, non-JSON and empty bodies,3.0as an integer,1.0against1, most-specific-wins, type lists; and, on a document written for the purpose,constthrough a chain of references,enum,oneOf/allOf/anyOf, array items, booleans vs their spelling, an unknown type name, and a reference into the middle of a schema being given the benefit of the doubt.AsyncApiFitnessTest(19): every DTO field the driver is told; unique correlation ids across evaluations; result and error as distinct targets; undeclared reply and silence as the two faults, and neither reported without--useExperimentalOracles; an unrecognised reply still covering its own target when they are off; fire-and-forget covered byPUBLISHED; headers as a map without the stamped one; a correlation id located in the payload; a topic from a protocol binding overriding the channel address; the configured timeout; a reply address announced at run time being published without waiting; a driver that could not publish, with and without saying why; a reply without its correlation id recorded rather than judged.AsyncApiStructureMutatorTest(5): the two bounds above, the maximum being reachable, added messages coming from the document, nothing happening when only one message is allowed. The first fails against RPC's arithmetic.AsyncApiModuleTest(2): a whole MIO search, 100 action evaluations, black-box, through the moduleMainbinds, against a stand-in that answers like the NCS service. Asserts that every operation was answered, that some operation reached both its declared replies, that nothing was a fault, and that no correlation id repeated.All AsyncAPI suites plus the config ones: 205 tests, 0 failures. jacoco over the AsyncAPI package: every new class above 90% line coverage.